Image analysis method, apparatus, program, and learned deep learning algorithm

ABSTRACT

The present invention provides an image analysis method for generating data indicating a region of a cell nucleus in an image of a tissue or a cell. The image analysis method is a method for analyzing an image of a tissue or a cell using a deep learning algorithm of a neural network structure, and generates data indicating a region of a cell nucleus in an analysis target image by the deep learning algorithm by generating analysis data from the analysis target image including an analysis target tissue or cell, and inputting the analysis data in the deep learning algorithm.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/193,422, filed on Nov. 16, 2018, entitled “IMAGE ANALYSIS METHOD,APPARATUS, PROGRAM, AND LEARNED DEEP LEARNING ALGORITHM,” which in turnclaims priority from Japanese Patent Application Publication No.2017-222178, filed on Nov. 17, 2017, entitled “IMAGE ANALYSIS METHOD,APPARATUS, PROGRAM, AND LEARNED DEEP LEARNING ALGORITHM”, the entirecontents of each of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to an image analysis method, apparatus,program, and learned deep learning algorithm. More specifically, theinvention relates to an image analysis method including the generationof data indicating the region of a cell nucleus at an optional positionof the image of the tissue or cell.

BACKGROUND

Japanese Patent Application Publication No. 2010-203949 discloses animage diagnosis support apparatus that determines and classifies atissue image in a pathological tissue image into four groups of normal,benign tumor, precancerous state, and cancer state. The imageclassifying means extracts the focus region from the image data,calculates a feature amount indicative of the feature of the focusregion, and classifies the group based on the calculated feature amount.Feature amounts are the density of clusters per unit area in the cellnucleus, the density of cluster areas, the area of clusters, thethickness of clusters, and the length of clusters. The image determiningmeans learns the relationship between the feature amount and thedetermination result and makes a determination based on the learnedlearning parameter. Learning executes machine learning using learningalgorithms such as support vector machines.

SUMMARY OF THE INVENTION

When definitively diagnosing whether a tumor is a malignant tumor,histopathological diagnosis using a histopathological sample isperformed. Histopathological diagnosis is often performed as anintraoperative rapid diagnosis to determine the site of excision oftissue containing malignant tumor during surgery. Intraoperative rapiddiagnosis is performed while the affected area of the patient is incisedand the surgical operation is temporarily halted awaiting adetermination of histopathological tissue diagnosis such as whether thetumor is malignant, whether there is tumor remains in the resectionmargin of the excised tissue, whether there is lymph node metastasis andthe like. The result of intraoperative rapid diagnosis determines thesubsequent direction of temporarily halted surgery of the patient.

Although the histopathological diagnosis is performed by a physician,particularly a pathologist, by observing the tissue sample with amicroscope or the like to diagnose the tissue sample, in order to beable to perform accurate definitive diagnosis by observing the tissuesample, the pathologist must repeatedly observe tissue samples ofvarious cases under supervision of a highly skilled pathologist, suchthat the training of a pathologist takes an extraordinary amount oftime.

There is a serious pathologist shortage, and as a result of thispathologist shortage, there is a delay in the confirmation of thediagnosis of a patient's malignant tumor, a delay in the start oftreatment, or the situation in which treatment is started withoutwaiting for the definitive diagnosis. Since both normal tissue diagnosisand rapid intraoperative diagnosis both rely on insufficiently fewpathologists, the workload of the individual pathologist becomesenormous and the labor conditions of the pathologist himself alsobecomes a problem. However, at present, no solution to this problem hasbeen found.

Therefore, it is considered that enabling a device to supportpathological tissue diagnosis will greatly contribute to the eliminationof the shortage of pathologists and the improvement of the laborconditions of pathologists, especially as the diagnosis is closer todetermination by the human eye.

In view of the fact that the apparatus supports pathological tissuediagnosis in the invention described in the above-mentioned JapanesePatent Application Publication No. 2010-203949, the pathology of thesample tissue is determined based on image analysis by machine learning.In this method, it is necessary to create a feature amount by humanhand. There is a problem that the ability of the person to greatlyinfluence the performance of the image analysis in the method ofcreating the feature amount by human hand.

For example, in tissue diagnosis or cell diagnosis using a microscope,one observation target is the state of the cell nucleus, and malignanttumors and benign tumors are differentiated from the size and form ofeach cell nucleus, the arrangement state of a plurality of cell nucleiand the like. Therefore, it is very important that the cell nucleus canbe accurately extracted in pathological tissue diagnosis as the basis ofhistological diagnosis and cell diagnosis.

The present invention provides an image analysis method for generatingdata indicating the cell nucleus region in images of tissue or cells.

One aspect of the present invention is an image analysis method. In thisaspect, the image analysis method for analyzing an image of a tissue ora cell using a deep learning algorithm (60) of a neural networkstructure includes generating analysis data (80) from the analysistarget image (78) including the tissue or cell to be analyzed (S21 toS23), inputting the analysis data (80) in the deep learning algorithm(60) (S24), and generating data (82, 83) indicating the tumorigenicstate of tissue or cells in the analysis target image (78) by the deeplearning algorithm (60) (S25 to S28). In this way it is possible togenerate of data indicating the region of a cell nucleus at an optionalposition of the image of a tissue or a cell.

It is preferable that the image to be analyzed is an image of a tissuediagnostic specimen and the analysis target image (78) contains a hueconsisting of one primary color or contains hues (R, G, B) combining twoor more primary colors.

It is preferable that the image to be analyzed is an image of a celldiagnostic specimen and the analysis target image (78) contains a hueconsisting of one primary color or contains hues (R, G, B) combining twoor more primary colors.

It is preferable that the data (82, 83) indicating the tumorigenic stateare data for distinguishing and presenting the nuclear region of a celland other regions.

It is preferable that the data (82, 83) indicating the tumorigenic stateare data for indicating boundary of the cell nucleus region and otherregions.

The deep learning algorithm (60) preferably determines whether anarbitrary position in the analysis target image (78) is a cell nucleusregion.

It is preferable to generate a plurality of analysis data (80) for eachregion of a predetermined number of pixels for one analysis target image(78). In this way it possible to improve the discrimination accuracy ofthe neural network (60).

The analysis data (80) preferably are generated for each region of apredetermined number of pixels including peripheral pixelscircumscribing a predetermined pixel, and the deep learning algorithm(60) preferably generates a label indicating whether a region is a cellnucleus in the predetermined pixel from the input analysis data (80). Inthis way it possible to improve the discrimination accuracy of theneural network (60).

It is preferable that the number of nodes of the input layer (60 a) ofthe neural network (60) corresponds to the product of the number ofcombined primary colors with the predetermined number of pixels of theanalysis data (80). In this way it possible to improve thediscrimination accuracy of the neural network (60).

It is preferable that the sample is a stained sample, and the analysistarget image (78) is an image obtained by imaging the stained sampleunder a bright field microscope.

The training data (74) used for learning of the deep learning algorithm(60) include a stained image (70) of a sample prepared by applying abright field observation stain to a sample of a tissue collected from anindividual or a sample containing cells collected from the individualcaptured in the bright field of the microscope and a fluorescence imageof cell nuclei (71) prepared by fluorescence staining of cell nuclei inthe same sample or a corresponding sample under fluorescence observationvia a fluorescence microscope, the training data (74) preferably beinggenerated based on the position in the sample of the fluorescence image(71) corresponding to the position in the obtained bright field image(70).

The stain used for bright field observation is preferably hematoxylinnucleus stain.

When the sample is a tissue sample, the bright field observation stainis preferably hematoxylin-eosin stain, and when the sample is a samplecontaining cells, the bright field observation stain is preferably aPapanicolaou stain.

The training data (74) preferably includes a label value indicating aregion of the cell nucleus extracted from the bright field image (70)and the fluorescence image (71). In this way it becomes possible for theneural network (50) to learn the label value indicating the region ofthe cell nucleus.

The training data (74) preferably includes a label value for each pixelof the bright field image (70). In this way it becomes possible for theneural network (50) to learn the label value indicating the region ofthe cell nucleus.

The training data (74) are preferably generated for each area of apredetermined number of pixels in the bright field image (70). In thisway it becomes possible to cause the neural network (50) to learn thelabel value indicating the region of the cell nucleus with highaccuracy.

The deep learning algorithm (60) preferably classifies the analysis data(80) into classes indicating regions of cell nuclei contained in theanalysis target image (78). In this way it is possible to classifyarbitrary positions of an analysis target tissue image and an imageincluding a cells into a region of a cell nucleus and another region.

It is preferable that the output layer (60 b) of the neural network (60)is a node having a soft max function as an activation function. In thisway it is possible for the neural network (60) to classify arbitrarypositions of an analysis target tissue image and an image including acell into a finite number of classes.

Each time the analysis data (80) are input, the deep learning algorithm(60) generates data (82) indicating whether the region is a cell nucleusregion included in the analysis target image (78) for each unit pixel.In this way it is possible to classify into a cell nucleus region andother regions for each unit pixel (one pixel) of an analysis targettissue image or an image including a cell.

It is preferable that the deep learning algorithm (60) is generatedaccording to the type of tissue sample or the type of sample containingcells. In this way it is possible to selectively use the deep learningalgorithm (60) according to the type of the analysis target tissue imageor the image including cells, and it is possible to improve thediscrimination accuracy of the neural network (60).

The analysis data (80) preferably are processed using a deep learningalgorithm (60) corresponding to the type of sample selected from aplurality of deep learning algorithms (60) according to the type of thetissue sample or the type of the sample including the cells. In this wayit is possible to selectively use the deep learning algorithm (60)according to the type of the analysis target tissue image or the imageincluding cells, and it is possible to improve the discriminationaccuracy of the neural network (60).

One aspect of the present invention is an image analysis apparatus. Inthis aspect, the image analyzing apparatus (200A) is an image analyzingapparatus for analyzing an image of a tissue or a cell using a deeplearning algorithm (60) of a neural network structure, and includes aprocessing unit (20A) for generating data (82, 83) indicating a regionof the cell nucleus in the analysis target image (78) via the deeplearning algorithm (60) by generating analysis data (80) from ananalysis target image (78) including an analysis target tissue or cell,and inputting the analysis data (80) in the deep learning algorithm(60). In this way it is possible to generate data indicating whether anarbitrary position of a tissue image or an image including a cell is aregion of a cell nucleus.

One aspect of the invention is a computer program. In this aspect, thecomputer program is a computer program for analyzing an image of atissue or a cell using a deep learning algorithm (60) of a neuralnetwork structure, the computer program causing a computer to execute aprocess to generate analysis data (80) from an analysis target image(78) including analysis target tissue or cells, a process to input theanalysis data (80) in the deep learning algorithm (60), and a process togenerate data (82, 83) indicating the region of a cell nucleus in theanalysis target image (78) by the deep learning algorithm (60). In thisway it is possible to generate of data indicating the region of a cellnucleus at an optional position of the image of a tissue or a cell.

One aspect of the present invention is a method of manufacturing alearned deep learning algorithm. In this aspect, the method ofmanufacturing the learned deep layer learning algorithm (60) includes afirst acquisition step of acquiring the first training data (72 r, 72 g,72 b) corresponding to the first training image (70) of imaged tissue orcells, a second acquisition step of acquiring second training data (73)corresponding to a second training image (71) indicating a region of thecell nucleus in the first training image (70), and a learning step (S13to S19) for causing the neural network (50) to learn the relationshipbetween the first training data (72 r, 72 g, 72 b) and the secondtraining data (73). In this way it is possible to produce a deeplearning algorithm for generating data indicative of regions of the cellnucleus for arbitrary positions of images of tissues or cells.

The first training data (72 r, 72 g, 72 b) preferably is the input layer(50 a) of the neural network (50) and the second training data (73)preferably is the output layer (50 b) of the neural network.

It is preferable that a step (S11) to generate the first training data(72 r, 72 g, 72 b) from the first training image (70) is included beforethe first acquisition step, and a step (S12) to generate the secondtraining data (73) from the second training image (71) is includedbefore the second acquisition step. In this way it is possible toproduce a deep learning algorithm for generating data indicative ofregions of the cell nucleus for arbitrary positions of images of tissuesor cells.

The first training image (70) is a bright field image captured under abright field microscope of a stained image of a sample prepared byapplying a bright field observation stain to a tissue sample taken froman individual or a sample containing cells collected from theindividual, and the second training image (71) is a fluorescence imageof a sample prepared by applying fluorescent nucleus stain to the sampleunder fluorescence observation under a microscope, wherein the positionin the sample of the fluorescence image (71) corresponds to the positionin the sample of the acquired bright field image (70).

One aspect of the present invention is a learned deep learning algorithm(60). In this aspect, the learned deep learning algorithm (60) is a deeplearning algorithm that learns the first training data (72 r, 72 g, 72b) as the input layer (50 a) of the neural network (50), and learns thesecond training data as the output layer (50 b) of a neural network(50), wherein the first training data (72 r, 72 g, 72 b) are generatedfrom the first training image (70) of imaged tissue or cells, and thesecond training data (73) indicates the region of the cell nucleus inthe first training image.

According to the present invention, data indicating the region of thecell nucleus can be generated for any position in the image of thetissue or cell.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a summary of a deep learningmethod;

FIGS. 2A-2C show a schematic diagram illustrating the details oftraining data;

FIG. 3 is a schematic diagram illustrating a summary of an imageanalysis method;

FIG. 4 is a schematic structural diagram of an image analysis systemaccording to a first embodiment;

FIG. 5 is a block diagram showing a hardware configuration of avendor-side apparatus 100;

FIG. 6 is a block diagram showing a hardware configuration of a userapparatus 200;

FIG. 7 is a block diagram illustrating functions of the deep learningapparatus 100A according to the first embodiment;

FIG. 8 is a flowchart showing a procedure of a deep learning process.

FIGS. 9A-9C are schematic diagrams illustrating details of learning by aneural network;

FIG. 10 is a block diagram illustrating functions of the image analysisapparatus 200A according to the first embodiment;

FIG. 11 is a flowchart showing a procedure of image analysis processing;

FIG. 12 is a schematic configuration diagram of an image analysis systemaccording to a second embodiment;

FIG. 13 is a block diagram illustrating functions of the integrated typeimage analysis apparatus 200B according to the second embodiment;

FIG. 14 is a schematic configuration diagram of an image analysis systemaccording to a third embodiment;

FIG. 15 is a block diagram illustrating functions of the integrated typeimage analysis apparatus 100B according to the third embodiment;

FIGS. 16A-16C are binarized images created from a bright field image, afluorescence image, and a fluorescence image used for generatingtraining data in Example 1;

FIGS. 17A-17B are results of analyzing an image of a sample (HE stain)of a first gastric cancer tissue in Example 1;

FIGS. 18A-18B are results of analyzing an image of a sample (HE stain)of a second gastric cancer tissue in Example 1;

FIGS. 19A-19B are results of analysis of an image of sealed samples(Papanicolaou stain) of gastric cancer site in Example 2; and

FIGS. 20A-20B are results of analyzing an image of sealed samples(Papanicolaou stain) of a non-gastric cancer site in Example 2.

DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

Hereinafter, a summary and embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings. Notethat in the following description and drawings the same referencenumerals denote the same or similar constituent elements, and therefore,descriptions of the same or similar constituent elements are omitted.

The present invention relates to an image analysis method for analyzingan image of a tissue or a cell, and an image analysis method using adeep learning algorithm of a neural network structure.

In the present invention, an image of a tissue or a cell is an imageobtained from a tissue sample or a sample containing a cell. Samples oftissue samples or samples containing cells are taken from individuals.The individual is not particularly limited, but is preferably a mammal,more preferably a human. Whether an individual is alive or deceased whena sample is taken from the individual is irrelevant. The tissue is notlimited as long as it is present in the individual. Examples of tissuescollected from an individual include surgically excised tissue, biopsytissue and the like. Samples containing cells are not limited insofar asthey are taken from individuals. For example, sputum, pleural effusion,ascites, urine, cerebrospinal fluid, bone marrow, blood, cystic fluidand the like can be mentioned.

The sample is intended as a preparation to prepare a sample includingthe tissue sample or sample including cells to be process forobservation by microscope and the like. The sample can be preparedaccording to a known method. For example, in the case of a tissuesample, after tissue is collected from an individual, the tissue isfixed with a prescribed fixing solution (formalin fixative or the like),the fixed tissue is embedded in paraffin, and the paraffin-embeddedtissue is sliced. The sliced section is placed on a slide glass. Thesample is completed by subjecting the slide glass with slice to stainingfor observation with an optical microscope, that is, for bright fieldobservation, and performing prescribed sealing treatment. A typicalexample of a tissue sample is a tissue diagnostic sample (pathologicalspecimen), and the staining is hematoxylin-eosin (HE) staining.

For example, in the case of a sample containing cells, the cells in thesample are attached to a slide glass by centrifugation, smearing or thelike, fixed with a predetermined fixing solution (ethanol or the like),stained for bright field observation, and a predetermined sealingprocess is carried out to complete the sample. A typical example of asample containing cells is a sample for cytological diagnosis(cytological specimen), and staining is Papanicolaou staining. The celldiagnostic sample also includes an imprinted sample of the tissuecollected for the tissue sample.

Both HE staining and Papanicolaou staining are nuclear staining withhematoxylin. Hematoxylin is widely used as a nuclear stain in tissuecell staining (for example, immunostaining, lectin staining, sugarstaining, fat staining, collagen fiber staining and the like).Therefore, the invention can be applied to all samples using hematoxylinfor nuclear staining.

In the present invention, two types of training images are used duringdeep learning. One of the images for training (first training image) isan image including tissues or cells contained in a sample of a tissuespecimen taken from an individual or a sample of a specimen containingcells taken from an individual. This image is obtained from samplesstained so that the tissue structure or cell structure can be recognizedby microscopic observation. Although the stain is not limited insofar asthe tissue structure or the cell structure can be recognized, a stainfor bright field observation is preferable. The above-described brightfield observation staining is not limited insofar as at least the cellnucleus and a part other than the cell nucleus can be stained so as tobe distinguishable by hue. When the sample is a mammalian tissue sample,for example, HE staining can be mentioned. For example, when the sampleis a sample containing mammalian cells, Papanicolaou stain can bementioned.

The next training image (the second training image) indicates which partof the first training image is the cell nucleus region, that is, theimage indicates a region in the first training image that correctlyidentifies the “cell nucleus region”. This image is obtained byselectively irradiating a cell nucleus of a sample that is the samesample from which the first training image was acquired or a samplecorresponding to the sample from which the first training image wasacquired (for example, a consecutively sliced sample), and the image iscaptured after performing fluorescent nuclear staining. The fluorescentnucleus stains includes, but is not limited to, 4′,6-diamidino-2-phenylindole (DAPI) stain.

In the example shown in FIG. 1, the first training image 70 of thetissue subjected to HE staining is DAPI-stained tissue and is used asthe first training image, and a second training image 71 from a positionin the sample corresponding to the position in the sample shown in thefirst training image is used as the second training image. An analysistarget image 78 of a tissue which has been subjected to the same brightfield staining as the first training image shown in FIG. 3 is used asanalysis data of the analysis target to be used during the imageanalysis process. The determination target to be learned as the correctinterpretation in the neural network 50 is the region of the cellnucleus contained in the tissue sample or the sample containing cells.

A case in which the region of the cell nucleus contained in an image ofthe HE stained tissue sample is determined by a deep learning algorithmis described as an example in the summary and the embodiment of theinvention.

Summary of Deep Learning Method and Image Analysis Method

First, a summary of the deep learning method and image analysis methodwill be described below. Next, each of the plurality of embodiments ofthe present invention will be described in detail.

Summary of Deep Learning Method

As shown in FIG. 1, training data generated from each of theabove-described first training image and second training image are usedin the deep learning method. Since the first training image is imaged asa color image, for example, under observation of the bright fieldmicroscope, the HE-stained sample includes a plurality of hues in thefirst training image.

The first training image (bright field image) 70 can be acquired inadvance using an image acquiring device such as a known opticalmicroscope, fluorescent microscope, or virtual slide scanner, forexample. Illustratively, in this embodiment it is preferable that colorimaging acquired from the image acquiring device is 24-bit color spacewith RGB. For the 24-bit color of RGB, it is preferable to express therespective densities (color densities) of red, green and blue with 8bits (256 levels). The first training image (bright field image) 70 maybe an image including one or more primary colors.

In the present invention, the hue is illustratively defined by acombination of the three primary colors of light or a combination of thethree primary colors of the color. The first training data are generatedfrom the first training image 70 by separating the hues appearing in thefirst training image 70 into individual primary colors, generating datafor each primary color, such that the first training data are datarepresented by a code corresponding to the density. In FIG. 1, singlecolor images 72R, 72G, and 72B separated for each primary color of red(R), green (G), and blue (B) which are the three primary colors oflight.

When the color density of each color is encoded for each pixel on thesingle color images 72R, 72G, and 72B, the entire image is encoded foreach image of R, G, and B by the encoding diagrams 72 r, 72 g, and 72 b.The color density also may be encoded with numerical values indicating256 gradations of each color. The color density also may bepre-processed for numerical values indicating 256 gradations of eachcolor, and the color density of each pixel may be encoded with numbersindicated by eight levels from 0 to 7, for example. The color densityencoding diagrams 72 r, 72 g, and 72 b in the single color image of eachcolor of R, G, and B shown in the example of FIG. 1 represent the colordensity in each pixel at eight gradations of values from 0 to 7 (threetones expressing tone). Symbols indicating color densities are alsoreferred to as color density values in this specification.

The second training image 71 is an image obtained by capturingfluorescent nuclei-stained samples by fluorescence observation under afluorescence microscope and capturing images of gray scales of two ormore gradations or color images. The second training image 71 may beacquired in advance, for example, using a known bright field imageacquisition device such as a fluorescence microscope or a virtual slidescanner.

The second training data are a true value image 73 showing the region ofthe cell nucleus of the learning target tissue, which is generated fromthe second training image 71 obtained by imaging the learning targettissue. The first training image 70 and the second training image 71 areimages obtained by imaging the same region or corresponding region ofthe tissue on the sample.

In the second training data, the second training image 71 of color orgrayscale having two or more gradations is converted into data as amonochrome fluorescence image by a binarization process, and thenlearned as the correct interpretation by the neural network 50 andresulting in a true image 73. When the target discriminated by theneural network 60 is a region of a cell nucleus, the true image 73 aredata indicating a region of the cell nucleus, that is, a correctinterpretation. The region of the cell nucleus and the other regions aredistinguished, and the region of the cell nucleus is discriminated bybinarizing the second training image 71. Whether the region is a cellnucleus region or a region other than the cell nucleus is determined,for example, by comparing the color density of each pixel in the imagewith a predetermined condition (for example, a color density thresholdvalue).

In the deep learning method, the color density encoded diagrams 72 r, 72g, and 72 b (also referred to as first training data) and the true image73 (also referred to as second training data) are used as the trainingdata 74, the neural network 50 learns the training data 74 with thecolor density encoded diagrams 72 r, 72 g, and 72 b as the input layer50 a and the true image 73 as the output layer 50 b. That is, a pair ofthe color density encoded diagrams 72 r, 72 g, and 72 b for each colorof R, G, and B and the true value image 73 are used as the training data74 for learning of the neural network 50.

A method of generating the training data 74 will be described withreference to FIGS. 2A to 2C. The training data 74 are data obtained bycombining the color density encoded diagrams 72 r, 72 g, and 72 b ofeach color of R, G, B and the true value image 73. In FIG. 2A, the imagesize (the size per training datum) of the training data 74 has beensimplified for convenience of explanation, and the color density encodeddiagrams 72 r, 72 g, 72 b and the true value image 73 have a total of 81pixels including 9 pixels in the vertical direction and 9 pixels in thehorizontal direction.

FIG. 2B shows an example of pixels configuring the training data 74.Three values 74 a shown in the upper part of FIG. 2B are density valuesof R, G, B in each pixel. Illustratively, the three values are stored inthe order red (R), green (G) and blue (B). Each pixel of the colordensity encoded diagrams 72 r, 72 g, and 72 b is shown in eight levelsof color density values from value 0 to value 7. This is a process ofconverting the brightness of each color image 72R, 72G, 72B representedin 256 steps to the eight-step color density value, respectively, as anexample of image preprocessing. For the color density value, forexample, the lowest brightness (a gradation group having a lowbrightness value when represented by 256 RGB colors) is set as the colordensity value 0, and gradually higher values are assigned as the degreeof brightness increases, with the highest brightness (gradation grouphaving high brightness value when expressed in RGB color of 256gradations) is set as color density value 7. The value 74 b shown in thelower part of FIG. 2B is binary data of the true image 73. The binarydata 74 b of the true image 73 is also called a label value. Forexample, the label value 1 indicates the region of the cell nucleus, andthe label value 0 indicates the other region. That is, in the true valueimage 73 shown in FIG. 1, the position of the label value changing from1 to 0 or the position of the pixel changing from 0 to 1 corresponds tothe boundary between the region of the cell nucleus and the otherregion.

The training data 75 shown in FIG. 2C are data obtained by extracting anarea of a predetermined number of pixels (hereinafter referred to as“window size”) of the training data 74 shown in FIG. 2A. Although thewindow size of the tumor site training data 75 is simplified to 3×3pixels for the sake of convenience of explanation, the actual preferablewindow size is illustratively about 113×113 pixels, and among them, asize of a 3×3 nuclei of normal gastric epithelial cells is preferablefrom the viewpoint of learning efficiency. For example, as shown in FIG.2C, a window W1 of 3×3 pixels is set and the window W1 is moved relativeto the training data 74. The center of the window W1 is located at anypixel of the training data 74, for example, the training data 74 in thewindow W1 indicated by the black frame are extracted as the trainingdata 75 of the window size. The extracted window size training data 75are used for learning of the neural network 50 shown in FIG. 1.

As shown in FIG. 1, the number of nodes of the input layer 50 a of theneural network 50 is the number of pixels of the training data 75 of theinput window size and the number of the primary colors included in theimage (for example, three nodes: R, G, and B in the case of the threeprimary colors of light). The color density value data 76 of each pixelof the window size training data 75 are set as the input layer 50 a ofthe neural network, and the binarized data 77 of the pixel located atthe center among the binary data 74 b corresponding to the true valueimage 73 of each pixel of the training data 75 is set as the outputlayer 50 b of the neural network 50 and are learned by the neuralnetwork 50. The color density value data 76 of each pixel are aggregatedata of the color density values 74 a of each color of R, G, B of eachpixel of the training data 75. As an example, when the training data 75has a window size of 3×3 pixels, one color density value 74 a is givenfor each R, G, and B for each pixel, such that the number of colordensity values of the color density value data 76 is “27” (3×3×3=27) andthe number of nodes of the input layer 50 a of the neural network 50also becomes “27”.

In this way the training data 75 of the window size input to the neuralnetwork 50 can be automatically created by the computer without beingcreated by the user. Efficient deep layer learning of the neural network50 is promoted in this way.

As shown in FIG. 2C, in the initial state the center of the window W1 islocated at the upper left corner of the training data 74. Thereafter,the training data 75 of the window size is extracted of the window W1,and the position of the window W1 is moved each time learning of theneural network 50 is performed. More specifically, the window W1 ismoved in units of one pixel so that the center of the window W1 scansall the pixels of the training data 74, for example. In this way thetraining data 75 of the window size extracted from all the pixels of thetraining data 74 are used for learning of the neural network 50.Therefore, the degree of learning of the neural network 50 can beimproved, and a deep learning algorithm having the structure of theneural network 60 shown in FIG. 3 is obtained as a result of deeplearning.

Image Analysis Method Summary

In the image analysis method shown in FIG. 3, analysis data 80 aregenerated from an analysis target image (bright field image) 78 obtainedby imaging a sample including a tissue or cells to be analyzed. Thesample preferably is stained the same as the first training image. Theanalysis target image 78 also can be acquired as a color image, forexample, using a known microscope, a virtual slide scanner or the like,for example. The analysis target image (bright field image) 78 may be animage including one or more primary colors. When color analysis targetimage 78 is encoded with color density values of each color of R, G, andB for each pixel, it is possible to express the entire image as anencoded diagram of color density values in each pixel of each R, G, B(analysis color density encoded diagrams 79 r, 79 g, 79 b). Colordensity encoded diagrams 79 r, 79 g, and 79 b indicating the codes ofthe color densities in the single color image of each color of R, G, andB shown in the example of FIG. 3 are obtained by substituting the threeprimary colors of images 79R, 79G, 79B with color density valuesrepresented by codes displayed in eight levels from 0 to 7.

The analysis data 80 are data obtained by extracting regions (that is,window sizes) having a predetermined number of pixels of the colordensity encoded diagrams 79 r, 79 g, and 79 b, and the data of thetissue or cell included in the analysis target image 78 include colordensity values. Although the analysis data 80 has a simplified windowsize of 3×3 pixels for convenience of explanation similar to thetraining data 75, a preferable window size is, for example, about113×113 pixels, and a size of a 3×3 nucleus of normal gastric epithelialcells is preferable in view of discrimination accuracy, for example, afield of view of 40 times which is about 113×113 pixels. For example, awindow W2 of 3×3 pixels is set, and the window W2 is moved relative tothe color density encoded diagrams 79 r, 79 g, and 79 b. The center ofthe window W2 is located at any pixel of the color density encodeddiagrams 79 r, 79 g, 79 b, and the color density encoded diagrams 79 r,79 g, 79 b are displayed in a window W2 indicated by a black frame of3×3 pixels to obtain window size analysis data 80. In this way theanalysis data 80 are generated for each area including peripheral pixelsaround the predetermined pixel from the color density encoded diagrams79 r, 79 g, and 79 b. The predetermined pixel means the pixel of thecolor density encoded diagrams 79 r, 79 g, and 79 b located at thecenter of the window W2, and the peripheral pixels are pixels within thewindow size range centered on the predetermined pixel of color densityencoded diagrams 79 r, 79 g, 79 b. In the analysis data 80, the colordensity values also are stored in the order of red (R), green (G) andblue (B) for each pixel similarly to the training data 74.

In the image analysis method, analysis data 80 are processed using adeep learning algorithm 60 having a neural network learned using thewindow size training data 75 shown in FIG. 1. Data 83 indicating whethera region is a cell nucleus region in the analysis target tissue or cellare generated by processing the analysis data 80.

Referring again to FIG. 3, the analysis data 80 cut out from the colordensity encoding diagrams 79 r, 79 g, and 79 b of the respective colorsR, G, and B are input to the neural network 60 configuring the deeplearning algorithm. The number of nodes of the input layer 60 a of theneural network 60 corresponds to the product of the number of inputpixels and the number of primary colors included in the image. When thecolor density value data 81 of each pixel of the analysis data 80 areinput to the neural network 60, an estimated value 82 (binary) of thepixel located at the center of the analysis data 80 is output from theoutput layer 60 b. For example, when the estimated value is 1, the valueindicates the region of the cell nucleus, and when the estimated valueis 0, it indicates the other region. That is, the estimated value 82output from the output layer 60 b of the neural network 60 is datagenerated for each pixel of the analysis target image and is dataindicating whether it is a cell nucleus region in the analysis targetimage. The estimated value 82 differentiates between the region of thecell nucleus and the other region by, for example, the value 1 and thevalue 0. The estimate value 82 is also called a label value and is alsocalled a class in the description of the neural network in the followingdescription. The neural network 60 generates a label indicating whetherthe region is a cell nucleus region relative to the pixel located at thecenter of the analysis data 80 among the input analysis data 80. Inother words, the neural network 60 classifies the analysis data 80 intoclasses indicating the regions of cell nuclei contained in the analysistarget image. Note that the color density value data 81 of each pixel isaggregate data of the color density values of each color of R, G, B ofeach pixel of the analysis data 80.

Thereafter, the analysis data 80 are extracted by window size whilemoving the window W2 by one pixel unit so that the center of the windowW2 scans all pixels of the color density encoded diagrams 79 r, 79 g,and 79 b of the respective colors R, G, B. The extracted analysis data80 are input to the neural network 60. In this way binary data 83 areobtained as data indicating whether or not the region is a cell nucleusregion in the analysis target image. In the example shown in FIG. 3, thecell nucleus region detection process also is performed on the binarydata 83 to obtain a cell nucleus region weighted image 84 indicating aregion of the cell nucleus. Specifically, the tumor cell nucleus regiondetection process is, for example, a process of detecting a pixel whoseestimate value 82 is a value 1, and is a process of actually identifyingthe region of the cell nucleus. The cell nucleus region weighted image84 is an image in which the region of the cell nucleus obtained by theimage analysis process is displayed superimposed on the analysis targetimage 78. After determining the region of the cell nucleus, a process ofdisplaying the cell nucleus and the other regions on the display deviceso as to be distinguishable also may be performed. For example,processing is performed such as filling the region of the cell nucleuswith color, drawing a line between the region of the tumor cell nucleusand the other region and the like so as to be distinguishably displayedon the display device.

First Embodiment

In the first embodiment, the configuration of a system that implementsthe deep learning method and image analysis method described in theabove outline will be specifically described.

Structure Summary

Referring to FIG. 4, the image analysis system according to the firstembodiment includes a deep learning apparatus 100A and an image analysisapparatus 200A. The vendor side apparatus 100 operates as the deep layerlearning apparatus 100A and the user side apparatus 200 operates as theimage analysis apparatus 200A. The deep learning apparatus 100A learnsusing the training data in the neural network 50 and provides the userwith a deep learning algorithm 60 that is trained with the trainingdata. The deep learning algorithm configured by the learned neuralnetwork 60 is provided from the deep learning apparatus 100A to theimage analysis apparatus 200A through a recording medium 98 or a network99. The image analysis apparatus 200A analyzes an analysis target imageusing a deep learning algorithm configured by the learned neural network60.

The deep layer learning apparatus 100A is configured by, for example, ageneral-purpose computer, and performs a deep learning process based ona flowchart to be described later. The image analysis apparatus 200A isconfigured by, for example, a general-purpose computer, and performsimage analysis processing based on a flowchart to be described later.The recording medium 98 is a computer readable non-transitory tangiblerecording medium such as a DVD-ROM or a USB memory.

The deep learning apparatus 100A is connected to an imaging device 300.The imaging device 300 includes an imaging element 301 and afluorescence microscope 302, and captures bright field images andfluorescence images of a learning sample 308 set on a stage 309. Thelearning sample 308 is subjected to the staining described above. Thedeep learning apparatus 100A acquires the first training image 70 andthe second training image 71 captured by the imaging device 300.

The image analysis apparatus 200A is connected to the imaging device400. The imaging device 400 includes an imaging element 401 and afluorescence microscope 402, and captures a bright field image of ananalysis target sample 408 set on the stage 409. The analysis targetsample 408 is stained in advance as described above. The image analysisapparatus 200A acquires the analysis target image 78 captured by theimaging device 400.

A known fluorescence microscope, a virtual slide scanner or the likehaving a function of imaging a sample can be used as the imaging devices300 and 400. The imaging device 400 also may be an optical microscopeinsofar as it has a function of imaging a sample.

Hardware Structure

Referring to FIG. 5, the vendor side apparatus 100 (100A, 100B) includesprocessing units 10 (10A, 10B), an input unit 16, and an output unit 17.

The processing unit 10 includes a CPU (Central Processing Unit) 11 thatperforms data processing to be described later, a memory 12 used as awork area for data processing, a recording unit 13 that records programsand processing data described later, a bus 14 for transmitting data, aninterface unit 15 for inputting and outputting data with an externaldevice, and a GPU (Graphics Processing Unit) 19. The input unit 16 andthe output unit 17 are connected to the processing unit 10.Illustratively, the input unit 16 is an input device such as a keyboardor a mouse, and the output unit 17 is a display device such as a liquidcrystal display. The GPU 19 functions as an accelerator for assistingarithmetic processing (for example, parallel arithmetic processing)performed by the CPU 11. That is, the process performed by the CPU 11 inthe following description means that the process includes a processperformed by the CPU 11 using the GPU 19 as an accelerator.

In order to perform the process of each step described below withreference to FIG. 8, the processing unit 10 pre-records the program andthe neural network 50 of the present invention in the recording unit 13before learning, for example, in an executable form. The execution formis, for example, a form generated by being converted from a programminglanguage by a compiler. The processing unit 10 performs processing usingthe program recorded in the recording unit 13 and the neural network 50before learning.

Unless otherwise specified in the following description, processingperformed by the processing unit 10 means processing performed by theCPU 11 based on the program stored in the recording unit 13 or thememory 12 and the neural network 50. The CPU 11 temporarily storesnecessary data (such as intermediate data being processed) with thememory 12 as a work area, and appropriately records data for long termstorage, such as calculation results, in the recording unit 13.

Referring to FIG. 6, the user side apparatus 200 (200A, 200B, 200C)includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and anoutput unit 27.

The processing unit 20 includes a CPU (Central Processing Unit) 21 forperforming data processing to be described later, a memory 22 used as awork area for data processing, a recording unit 23 for recordingprograms and processing data described later, a bus 24 for transmittingdata, an interface section 25 for inputting and outputting data with anexternal device, and a GPU (Graphics Processing Unit) 29. The input unit26 and the output unit 27 are connected to the processing unit 20.Illustratively, the input unit 26 is an input device such as a keyboardor a mouse, and the output unit 27 is a display device such as a liquidcrystal display. The GPU 29 functions as an accelerator for assistingarithmetic processing (for example, parallel arithmetic processing)performed by the CPU 21. That is, the process performed by the CPU 21 inthe following description means that the process includes a processperformed by the CPU 21 using the GPU 29 as an accelerator.

In order to perform the processes of each step described below withreference to FIG. 13, the processing unit 20 records in advance theprogram according to the present invention and the learned neuralnetwork structure deep layer learning algorithm 60, for example, in anexecutable format in the recording unit 23. The execution form is, forexample, a form generated by being converted from a programming languageby a compiler. The processing unit 20 performs processing using theprogram recorded in the recording unit 23 and the deep learningalgorithm 60.

Unless otherwise stated in the following description, the processingperformed by the processing unit 20 means the processing actuallyperformed by the processing unit 20 based on the program stored in therecording unit 23 or the memory 22 and the deep learning algorithm 60.The CPU 21 temporarily stores necessary data (such as intermediate databeing processed) with the memory 22 as a work area, and appropriatelyrecords data for long term storage, such as calculation results, in therecording unit 23.

Function Block and Processing Procedure

Deep Learning Process

Referring to FIG. 7, the processing unit 10A of the deep learningapparatus 100A according to the first embodiment includes a trainingdata generating unit 101, a training data input unit 102, and analgorithm updating unit 103. These functional blocks are realized byinstalling a program that causes a computer to execute a deep layerlearning process in the recording unit 13 or the memory 12 of theprocessing unit 10A and executing this program by the CPU 11. The windowsize database 104 and the algorithm database 105 are recorded in therecording unit 13 or the memory 12 of the processing unit 10A.

The first training image 70 and the second training image 71 of thesample for learning are captured in advance by the imaging device 300and stored in advance in the recording unit 13 or the memory 12 of theprocessing unit 10A. The neural network 50 is stored in advance in thealgorithm database 105 in association with, for example, the type (forexample, organization name) of the tissue sample from which the analysistarget sample is derived or the type of sample including cells.

The processing unit 10A of the deep learning apparatus 100A performs theprocess shown in FIG. 8. When describing each function block shown inFIG. 7, the processes of steps S11 to S13, S18 and S19 are performed bythe training data generating unit 101. The process of step S14 isperformed by the training data input unit 102. The processes of stepsS15 to S17 are performed by the algorithm updating unit 103.

In steps S11 to S19 described below, a deep learning process for a pairof a first training image 70 (bright field images) and a second trainingimage (second training image 71) will be described.

In step S11, the processing unit 10A generates color density encodeddiagrams 72 r, 72 g, and 72 b for R, G, B colors from the input firsttraining image 70. The color density encoded diagrams 72 r, 72 g, and 72b are created by stepwise expression of the color density values of therespective colors of R, G, and B of each pixel of the first trainingimage 70. In the present embodiment, color density encoding diagrams 72r, 72 g, and 72 b are created for each R, G, B gradation image with thecolor density value set to 8 levels from 0 to 7. Assignment of a colordensity value is carried out, for example, by setting the lowestbrightness as the color density value 0, gradually assigning a highervalue as the degree of brightness increases, and setting the highestbrightness as the color density value 7.

In step S12, the processing unit 10A binarizes the gradation of eachpixel of the input second training image 71 to generate a true valueimage 73. The true value image 73 (binarized image 73) is used togenerate training data to cause the neural network 50 to learn a correctinterpretation. The binarization process is performed, for example, bycomparing the gradation of each pixel in the image with a predeterminedthreshold value.

In step S13, the processing unit 10A receives input of the type oftissue for learning from the operator on the side of the deep learningapparatus 100A via the input unit 16. The processing unit 10A sets thewindow size by referring to the window size database 104 based on thetype of the input tissue and refers to the algorithm database 105 to setthe neural network 50 used for learning. In the present embodiment inwhich a stomach tissue sample analysis target, the window size is, forexample, 113×113 pixels. This pixel size is a size in an image capturedat, for example, 40 times magnification. Illustratively, it is a sizethat supports that the entire shape of the cell nucleus region of atleast one cell out of two to nine cells is included in the window. Thewindow size is a unit of training data to be input to the neural network50 at the time of one input, and the product of the number of pixels ofthe tumor site training data 75 of the window size and the number of theprimary colors included in the image corresponds to the number of nodesof the input layer 50 a. The window size is associated with the type ofthe tissue sample or the type of the sample including cells and recordedin advance in the window size database 104.

In step S14, the processing unit 10A generates the window size trainingdata 75 from the color density encoded diagrams 72 r, 72 g, and 72 b andthe true image 73. Specifically, in the “Summary of the Deep LearningMethod” as described above with reference to FIGS. 2A to 2C, thetraining data 75 of the window size is created by the window W1 from thetraining data 74 of the combined color density encoded diagrams 72 r, 72g, and 72 b and the true image

In step S15 shown in FIG. 8, the processing unit 10A learns the neuralnetwork 50 using the window size training data 75. The learning resultof the neural network 50 is accumulated each time the neural network 50learns using the window size training data 75.

In the image analysis method according to the embodiment, since aconvolutional neural network is used and the stochastic gradient descentmethod is used, in step S16, the processing unit 10A determines whetherlearning results for a predetermined number of trials are accumulated.The processing unit 10A performs the processing of step S17 when thelearning results are accumulated for a predetermined number of trials,and the processing unit 10A performs the processing of step S18 when thelearning results are not accumulated for a predetermined number oftrials.

When learning results have been accumulated for a predetermined numberof trials, in step S17 the processing unit 10A updates the couplingweight w of the neural network 50 using the learning results accumulatedin step S15. In the image analysis method according to the embodiment,since the stochastic gradient descent method is used, the couplingweight w of the neural network 50 is updated when the learning resultsfor a predetermined number of trials are accumulated. Specifically, theprocess of updating the coupling weight w is a process of performingcalculation by the gradient descent method shown in (Equation 11) and(Equation 12) described later.

In step S18, the processing unit 10A determines whether the specifiednumber of pixels in the input image have been processed. The input imageis the training data 74; when a series of processes from step S14 tostep S17 has been performed for the specified number of pixels in thetraining data 74, the deep learning process is terminated. Learning bythe neural network does not necessarily have to be performed for allpixels in the input image, and the processing unit 10A can performlearning by processing a part of pixels in the input image, that is, aprescribed number of pixels. The prescribed number of pixels also may beall pixels in the input image.

When the prescribed number of pixels in the input image have not beenprocessed, in a step S19 the processing unit 10A moves the centerposition of the window one pixel unit in the training data 74 as shownin FIG. 2C. Thereafter, the processing unit 10A performs a series ofprocesses from step S14 to step S17 at the new window position aftermovement. That is, in step S14 the processing unit 10A extracts thetraining data 74 in the window size at the new window position after themovement. Subsequently, in step S15, the processing unit 10A causes theneural network 50 to learn using the newly cut window size training data75. When learning results for a predetermined number of trials areaccumulated in step S16, the processing unit 10A updates the couplingweight w of the neural network 50 in step S17. Learning of the neuralnetwork 50 for each window size is performed for a specified number ofpixels in the training data 74.

The degree of learning of the neural network 50 is improved by repeatingthe deep learning processes of steps S11 to S19 for a pair of inputimage relative to a plurality of pairs of different input images. Inthis way a deep learning algorithm 60 of the neural network structureshown in FIG. 3 is obtained.

Neural Network Structure

In the first embodiment shown in FIG. 9A, a neural network of a deeplearning type is used. The neural network of the deep learning type isconfigured by an input layer 50 a, an output layer 50 b, and anintermediate layer 50 c between the input layer 50 a and the outputlayer 50 b, and the intermediate layer 50 c is configured by a pluralityof layers as in the neural network shown in FIG. 9A. The number oflayers configuring the intermediate layer 50 c, for example, may be fiveor more.

In the neural network 50, a plurality of nodes 89 arranged in layers areconnected between layers. In this way information propagates from theinput side layer 50 a to the output side layer 50 b only in onedirection indicated by an arrow D in the drawing. In the presentembodiment, the number of nodes of the input layer 50 a corresponds tothe product of the number of pixels of the input image, that is, thenumber of pixels of the window W1 shown in FIG. 2C and the number ofprimary colors included in each pixel. Since the pixel data (colordensity values) of the image can be input to the input layer 50 a, theuser can input the input image to the input layer 50 a withoutseparately calculating the feature amount from the input image.

Operation at Each Node

FIG. 9B is a schematic diagram showing the operation at each node. Eachnode 89 receives a plurality of inputs and calculates one output (z). Inthe example shown in FIG. 9B, the node 89 receives four inputs. Thetotal input (u) received by the node 89 is expressed by the following(Equation 1).

Function 1

u=w ₁ x +w ₂ x ₂ +w ₃ x ₃ +w ₄ x ₄ +b  (Equation 1)

Each input is multiplied by a different weight. In equation (1), b is avalue called bias. The output (z) of the node is an output of apredetermined function f relative to the total input (u) represented by(Equation 1), and is expressed by the following (Equation 2). Thefunction f is called an activation function.

Function 2

z=f(u)  (Equation 2)

FIG. 9C is a schematic diagram showing the operation between the nodes.In the neural network 50, the nodes that output the result (z)represented by (Equation 2) are arranged in a layered manner relative tothe total input (u) represented by (Equation 1). The output of the nodeof the previous layer becomes the input of the node of the next layer.In the example shown in FIG. 9C, the output of the node 89 a on thelayer on the left side in the figure becomes the input to the node 89 bon the layer on the right side in the figure. Each node 89 b of theright side layer receives an output from a node 89 a on the left sidelayer, respectively. Different weights are applied to each couplingbetween each node 89 a on the left side layer and each node 89 b on theright side layer. Output of each of the plurality of nodes 89 a on theleft side layer is designated x1˜x4, and the inputs to each of the threenodes 89 b on the right side layer are represented by (Equation 3-1) to(Equation 3-3) below.

Function 3

u ₁ =W ₁₁ X ₁ +W ₁₂ X ₂ +W ₁₃ x ₃ +W ₁₄ X ₄ +b ₁  (Equation 3-1)

u ₂ =w ₂₁ x ₁ +w ₂₂ x ₂ +w ₂₃ x ₃ +w ₂₄ x ₄ +b ₂  (Equation 3-2)

u ₃ =w ₃₁ x ₁ +w ₃₂ x ₂ +w ₃₃ x ₃ +w ₃₄ x ₄ +b ₃  (Equation 3-3)

Generalizing these (Equation 3-1) to (Equation 3-3) results in (Equation3-4). Here, i=1, . . . I, j=1, . . . J.

$\begin{matrix}{{Function}\mspace{14mu} 4} & \; \\{u_{j} = {{\sum_{i = 1}^{J}{w_{ji}x_{i}}} + b_{j}}} & \left( {{Equation}\mspace{14mu}\text{3-4}} \right)\end{matrix}$

Applying Equation 3-4 to the activation function yields the output. Theoutput is expressed by the following (Equation 4).

Function 4

z _(j) =f(u _(j)) (j=1,2,3)  (Equation 4)

Activation Function

In the image analysis method according to the embodiment, a rectifiedlinear unit function is used as the activation function. The rectifiedlinear unit function is expressed by the following (Equation 5).

Function 6

f(u)=max(u,0)  (Equation 5)

Equation 5 is a function that sets u=0 among u=0 in the linear functionof z =u. In the example shown in FIG. 9C, the output of the node withj=1 is expressed by the following equation according to Equation 5.

Function 7

z ₁=max((w ₁₁ x ₁ +w ₁₂ x ₂ +w ₁₃ x ₃ +w ₁₄ x ₄ +b ₁),0)

Neural Network Learning

Let y (x: w) be the function expressed using the neural network, thefunction y (x: w) changes as the parameter w of the neural networkchanges. Adjusting the function y (x: w) so that the neural networkselects a more suitable parameter w for the input x is referred to aslearning of the neural network. Suppose that multiple sets of inputs andoutputs of functions expressed using a neural network are given.Assuming that the desired output for an input x is d, the input/outputset is {(x₁, d₁), (x₂, d₂), . . . , (x_(n), d_(n))}. The set of each setrepresented by (x, d) is referred to as training data. Specifically, theset of a set of a color density values for each pixel and a label for atrue value image in a single color image of each color of R, G, B shownin FIG. 2B is the training data shown in FIG. 2A.

Learning by a neural network means that when input xn is given to anyinput/output pair (xn, dn), weight w is adjusted so that the output y(xn: w) of the neural network is as close as possible to the output dn.An error function is a measure for measuring the proximity between afunction expressed using a neural network and training data.

Function 8

y(x _(n) :w)≈d _(n)

The error function is also referred to as a loss function. The errorfunction E(w) used in the image analysis method according to theembodiment is represented by the following (Equation 6). Equation 6 isreferred to as cross entropy.

$\begin{matrix}{{Function}\mspace{14mu} 9} & \; \\{{E(w)} = {{- \Sigma_{n = 1}^{N}}\Sigma_{k = 1}^{K}d_{nk}{{\log y}_{k}\left( {x_{n};w} \right)}}} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

A method of calculating the cross entropy of (Equation 6) will bedescribed. In the output layer 50 b of the neural network 50 used in theimage analysis method according to the embodiment, that is, in the finallayer of the neural network, an activation function is used forclassifying the input x into a finite number of classes according to thecontent. The activation function is called a softmax function and isexpressed below (Equation 7). Note that it is assumed that the samenumber of nodes as the class number k are arranged in the output layer50 b. The total input u of each node k (k=1, . . . , K) of the outputlayer L is obtained from the output of the previous layer L−1 by uk(L)respectively. In this way the output of the kth node of the output layercan be expressed as follows (Equation 7).

$\begin{matrix}{{Function}\mspace{14mu} 10} & \; \\{{y_{k} \equiv z_{k}^{(L)}} = \frac{\exp\left( u_{k}^{(L)} \right)}{\Sigma_{j = 1}^{K}{\exp\left( u_{j}^{(L)} \right)}}} & \left( {{Equation}\mspace{14mu} 7} \right)\end{matrix}$

Equation 7 is a softmax function. The output y₁, . . . , y_(K) Is always1.

Each class is expressed as C₁, . . . , C_(K), and the output y of thenode k of the output layer L_(K)(that is, u_(k) ^((L))) indicates theprobability that given input x belongs to class C_(K). Please refer toEquation 8 below. The input x is classified into a class having themaximum probability represented by Equation 8.

Function 11

p(C _(k|x))=y _(k) =z _(k) ^((L))  (Equation 8)

In the learning of the neural network, the function expressed by theneural network is regarded as a model of the posterior probability ofeach class, and the likelihood of the weight w relative to the trainingdata under such a probability model is evaluated and a weight w thatmaximizes likelihood is selected.

The target output d_(n) by the softmax function of (Equation 7) is setto 1 only when the output is a correct class, and 0 if the output isotherwise. When the target output is expressed in vector formd_(n)=[d_(n1), . . . , d_(nK)], for example, when the correct class ofinput xn is C3, only the target output dn3 is 1, and the other targetoutputs are 0. When encoding in this manner, the posterior distributionis represented as follows (Equation 9).

$\begin{matrix}{{Function}\mspace{14mu} 12} & \; \\{{p\left( {d❘x} \right)} = {\Pi_{k = 1}^{K}{p\left( {C_{k}❘x} \right)}^{d_{k}}}} & \left( {{Equation}\mspace{14mu} 9} \right)\end{matrix}$

The likelihood L(w) of weight w relative to the training data {(xn, dn)}(n=1, . . . , N) is expressed below (Equation 10). The error function ofEquation 6 is derived by taking the logarithm of the likelihood L(w) andinverting the sign.

$\begin{matrix}{\mspace{79mu}{{Function}\mspace{14mu} 13}} & \; \\{{L(w)} = {{\Pi_{n = 1}^{N}{p\left( {{d_{n}❘x_{n}};w} \right)}} = {{\Pi_{n = 1}^{N}\Pi_{k = 1}^{K}{p\left( {C_{k}❘x_{n}} \right)}^{d_{nk}}} = {\Pi_{n = 1}^{N}{\Pi_{k = 1}^{K}\left( {y_{k}\left( {x;w} \right)} \right)}^{d_{nk}}}}}} & \left( {{Equation}\mspace{14mu} 10} \right)\end{matrix}$

Learning means minimizing the error function E(w) calculated based onthe training data for the parameter w of the neural network. In theimage analysis method according to the embodiment, the error functionE(w) is expressed by (Equation 6).

Minimizing the error function E(w) for the parameter w has the samemeaning as finding the local minima of the function E(w). The parameterw is the weight of the coupling between the nodes. The minimum point ofthe weight w is obtained by iterative calculation that iterativelyupdates the parameter w using an arbitrary initial value as a startingpoint. An example of such a calculation is the gradient descent method.

In the gradient descent method, a vector expressed by the followingEquation 11 is used.

$\begin{matrix}{{Function}\mspace{14mu} 14} & \; \\{{\nabla E} = {\frac{\partial E}{\partial w} = \left\lbrack {\frac{\partial E}{\partial w_{1}},\;{.\;.\;.}\;,\;\frac{\partial E}{\partial w_{M}}} \right\rbrack^{T}}} & \left( {{Equation}\mspace{14mu} 11} \right)\end{matrix}$

In the gradient descent method, the process of moving the value of thecurrent parameter win the negative gradient direction (that is, −∇E) isrepeated many times. When the current weight is designated w(t) and theweight after movement is w(t+1), the calculation by the gradient descentmethod is represented by the following Equation 12. The value t meansthe number of times the parameter w has been moved.

Function 15

w ^((t+1)) =w ^((t)) −∈∨ E  (Equation 12)

The symbol

Function 16

ϵ

is a constant that determines the magnitude of the update amount of theparameter w, and is referred to as a learning coefficient. By repeatingthe operation represented by (Equation 12), the error function E(w)(t))decreases as the value t increases, and the parameter w reaches theminimum point.

Note that the calculation according to Equation 12 may be performed onall the training data (n=N) or may be performed only on a part of thetraining data. The gradient descent method performed for only some ofthe training data is referred to as the stochastic gradient descentmethod. A stochastic gradient descent method is used in the imageanalysis method according to the embodiment.

Image Analysis Process

Referring to FIG. 10, the processing unit 20A of the image analysisapparatus 200A according to the first embodiment includes an analysisdata generation unit 201, an analysis data input unit 202, an analysisunit 203, and a tumor cell nucleus region detection unit 204. Thesefunctional blocks are realized by installing a program according to thepresent invention for causing a computer to execute an image analysisprocess in the recording unit 23 or the memory 22 of the processing unit20A and executing this program by the CPU 21. The window size database104 and the algorithm database 105 are provided from the deep learningapparatus 100A through the recording medium 98 or the network 99, andrecorded in the recording unit 23 or the memory 22 of the processingunit 20A.

The analysis target image 78 of the analysis target tissue is capturedin advance by the imaging device 400, and recorded in the recording unit23 or the memory 22 of the processing unit 20A in advance. The deeplearning algorithm 60 including the learned coupling weight w is storedin the algorithm database 105 in association with the type of tissuesample (for example, tissue name) from which the sample of the analysistarget tissue is derived or the type of sample including cells, andfunctions as a program module which is a part of a program that causes acomputer to execute an image analysis process. That is, the deeplearning algorithm 60 is used in a computer having a CPU and a memory,and causes the computer to function to execute computation or processingof specific information corresponding to the purpose of use, such asoutputting data indicating whether the region is the cell nucleus in theanalysis target tissue. Specifically, the CPU 21 of the processing unit20A performs the calculation of the neural network 60 based on thelearned coupling weight w in accordance with the algorithm prescribed inthe deep learning algorithm 60 recorded in the recording unit 23 or thememory 22. The CPU 21 of the processing unit 20A performs an operationon the captured analysis target image 78 of the analysis target tissuewhich is input to the input layer 60 a and output from the output layer60 b binary image 83 of the data indicating whether the region is a cellnucleus in the analysis target tissue.

Referring to FIG. 11, the processing unit 20A of the image analysisapparatus 200A performs the processes shown in FIG. 11. When describingeach function block shown in FIG. 10, the processes of steps S21 and S22are performed by the analysis data generation unit 201. The processes ofsteps S23, S24, S26, and S27 are performed by the analysis data inputunit 202. The processes of steps S25 and S28 are performed by theanalysis unit 203. The process of step S29 is performed by the cellnucleus region detection unit 204.

In step S21, the processing unit 20A generates color density encodeddiagrams 79 r, 79 g, and 79 b of R, G, and B colors from the inputanalysis target image 78. The generation method of the color densityencoded diagrams 79 r, 79 g, and 79 b is the same as the generationmethod of step S11 at the time of the deep learning process shown inFIG. 8.

In step S22 shown in FIG. 11, the processing unit 20A accepts the inputof the tissue type from the user on the image analysis apparatus 200Aside as the analysis condition through the input unit 26. The processingunit 20A refers to the window size database 104 and the algorithmdatabase 105 on the basis of the entered tissue type to set the windowsize used for analysis, and acquires the deep learning algorithm 60 usedfor analysis. The window size is a unit of analysis data to be input tothe neural network 60 at the time of one input, and the product of thenumber of pixels of the window size analysis data 80 and the number ofprimary colors included in the image corresponds to the number of nodesof input layer 60 a. The window size is associated with the tissue type,and stored in the window size database 104 in advance. The window sizeis, for example, 3×3 pixels as shown in the window W2 of FIG. 3. Thedeep learning algorithm 60 is also recorded in advance in the algorithmdatabase 105 shown in FIG. 10 in association with the type of the tissuesample or the type of the sample including cells.

In step S23 shown in FIG. 11, the processing unit 20A generates thewindow size analysis data 80 from the color density encoded diagrams 79r, 79 g, and 79 b.

In step S24, the processing unit 20A inputs the analysis data 80 shownin FIG. 3 to the deep learning algorithm 60. The initial position of thewindow is, for example, a position at the center of 3×3 pixels in thewindow corresponding to the upper left corner of the analysis targetimage, as in step S15 in the deep learning process. When the processingunit 20A inputs the data 81 of a total of 27 color density values of 3×3pixels×3 primary colors included in the window size analysis data 80 tothe input layer 60 a, the deep learning algorithm 60 outputsdetermination result 82 to the output layer 60 b.

In step S25 shown in FIG. 11, the processing unit 20A records thedetermination result 82 output to the output layer 60 b shown in FIG. 3.The determination result 82 is an estimated value (binary) of pixelslocated at the center of the color density encoded diagrams 79 r, 79 g,and 79 b, which are analysis targets. For example, when the estimatedvalue is 1, the value indicates the region of the cell nucleus, and whenthe estimated value is 0, it indicates the other region.

In step S26 shown in FIG. 11, the processing unit 20A determines whetherall the pixels in the input image have been processed. The input imageis the color density encoded diagrams 79 r, 79 g, and 79 b shown in FIG.3, and the process of step S28 is performed for all the pixels in thecolor density encoded diagrams 79 r, 79 g, and 79 b when the series ofprocesses from step S23 to step S25 shown in FIG. 11 have beenperformed.

When all the pixels in the input image have not been processed, in stepS27 the processing unit 20A moves the center position of the window W2by one pixel unit within the color density encoded diagrams 79 r, 79 g,and 79 b shown in FIG. 3 similarly to step S19 in the deep learningprocess. Thereafter, the processing unit 20A performs a series ofprocesses from step S23 to step S25 at the position of the new window W2after movement. In step S25, the processing unit 20A records thedetermination result 82 corresponding to the new window position afterthe movement. A binary image 83 of the analysis result is obtained byrecording the discrimination result 82 for each window size on all thepixels in the analysis target image. The image size of the binary image83 as the analysis result is the same as the image size of the analysistarget image. Here, in the binary image 83, the value 1 and the value 0of the estimate value may be numerical data attached to each pixel, andin place of the estimate value 1 and 0, for example, the value 1 and thevalue 0 may be displayed in a display color associated with each ofthem.

In step S28 shown in FIG. 11, the processing unit 20A outputs the binaryimage 83 of the analysis result to the output unit 27.

In step S29 following step S28, the processing unit 20A also performs acell nucleus region detection process on the cell nucleus region of thebinary image 83 of the analysis result. In the binary image 83, theregion of the cell nucleus and the other regions are distinguished andrepresented by binary values. Therefore, in the binary image 83, it ispossible to discriminate the region of the cell nucleus by detecting theposition of the pixel whose estimate value of the pixel changes from 1to 0 or the pixel changing from 0 to 1. As another embodiment, it alsois possible to detect the boundary between the region of the cellnucleus and the other region, that is, detect the region of the cellnucleus.

Although optional, the processing unit 20A may create a cell nucleusregion weighted image 84 by superimposing the region of the obtainedcell nucleus on the analysis target image 78. The processing unit 20Aoutputs the created cell nucleus region weighted image 84 to the outputunit 27, and ends the image analysis process.

As described above, the user of the image analysis apparatus 200A canacquire the binary image 83 as the analysis result by inputting theanalysis target image 78 of the analysis target tissue to the imageanalysis apparatus 200A. The binary image 83 represents the region ofthe cell nucleus and other regions in the analysis target sample, andthe user can discriminate the region of the cell nucleus in the analysistarget sample.

The user of the image analysis apparatus 200A also can acquire the cellnucleus region weighted image 84 as the analysis result. The cellnucleus region weighted image 84 is generated, for example, by fillingthe region of the cell nucleus with a color in the analysis target image78. In another embodiment, the cell nucleus region weighted image 84 isgenerated by overlapping a boundary line between a region of cellnucleus and another region. In this way the user can grasp the region ofthe cell nucleus at a glance in the analysis target tissue.

Showing the region of the cell nucleus in the analysis target samplehelps a person not familiar with the sample to understand the state ofthe cell nucleus.

Second Embodiment

Hereinafter, the image analysis system according to the secondembodiment will be described with respect to points different from theimage analysis system according to the first embodiment.

Structure Summary

Referring to FIG. 12, the image analysis system according to the secondembodiment includes a user side apparatus 200, and the user sideapparatus 200 operates as an integrated image analysis apparatus 200B.The image analysis apparatus 200B is configured by, for example, ageneral-purpose computer, and performs both of the deep learning processand the image analysis process described in the first embodiment. Thatis, the image analysis system according to the second embodiment is astand-alone system that performs deep learning and image analysis on theuser side. The image analysis system according to the second embodimentdiffers from the image analysis system according to the first embodimentin that the integrated type image analysis apparatus 200B installed onthe user side has the functions of both the deep learning apparatus 100Aand the image analysis apparatus 200A according to the first embodiment.

The image analysis apparatus 200B is connected to the imaging apparatus400. At the time of the deep learning process, the imaging apparatus 400acquires the first training image 70 and the second training image 71 ofthe learning tissue, and acquires the analysis target image 78 of theanalysis target tissue at the time of the image analysis process.

Hardware Structure

The hardware configuration of the image analysis apparatus 200B issimilar to the hardware configuration of the user side apparatus 200shown in FIG. 6.

Function Block and Processing Procedure

Referring to FIG. 13, the processing unit 20B of the image analysisapparatus 200B according to the second embodiment includes a trainingdata generation unit 101, a training data input unit 102, an algorithmupdate unit 103, an analysis data generation unit 201, an analysis datainput unit 202, an analysis unit 203, and a cell nucleus regiondetection unit 204. These functional blocks are realized by installing aprogram that causes a computer to execute a deep learning process and animage analysis process in the recording unit 23 or the memory 22 of theprocessing unit 20B and executing this program by the CPU 21. The windowsize database 104 and the algorithm database 105 are recorded in therecording unit 23 or the memory 22 of the processing unit 20B, and bothare used jointly during deep learning and image analysis processing. Thelearned neural network 60 is stored beforehand in the algorithm database105 in association with the type of tissue or the type of sampleincluding cells, the coupling weight w is updated by the deep learningprocess, and stored as the deep learning algorithm 60 in the algorithmdatabase 105. It should be noted that the first training image 70 andthe second training image 71 which are the first training images forlearning are captured in advance by the imaging apparatus 400 and storedin advance in the recording unit 23 or the memory 22 of the processingunit 20B. The analysis target image 78 of the analysis target sample isalso imaged in advance by the imaging apparatus 400 and recorded in therecording unit 23 or the memory 22 of the processing unit 20B inadvance.

The processing unit 20B of the image analysis apparatus 200B performsthe processing shown in FIG. 11 at the time of the deep learning processand the processing shown in FIG. 8 at the time of the image analysisprocess. When describing each function block shown in FIG. 13, theprocesses of steps S11 to S13, S18 and S19 are performed by the trainingdata generating unit 101 during the deep learning process. The processof step S14 is performed by the training data input unit 102. Theprocesses of steps S15 to S17 are performed by the algorithm updatingunit 103. The processes of steps S21 and S22 are performed by theanalysis data generation unit 201 at the time of image analysis process.The processes of steps S23, S24, S26, and S27 are performed by theanalysis data input unit 202. The processes of steps S25 and S28 areperformed by the analysis unit 203. The process of step S29 is performedby the cell nucleus region detection unit 204.

The procedure of the deep learning process and the procedure of theimage analysis process performed by the image analysis apparatus 200Baccording to the second embodiment are similar to the proceduresperformed by the deep learning apparatus 100A and the image analysisapparatus 200A according to the first embodiment. Note that the imageanalysis apparatus 200B according to the second embodiment differs fromthe deep learning apparatus 100A and the image analysis apparatus 200Aaccording to the first embodiment in the following points.

In step S13 at the time of the deep learning process, the processingunit 20B receives an input of the type of tissue for learning from theuser of the image analysis apparatus 200B via the input unit 26. Theprocessing unit 20B sets the window size by referring to the window sizedatabase 104 based on the type of the input tissue, and refers to thealgorithm database 105 to set the neural network 50 used for learning.

As described above, the user of the image analysis apparatus 200B canacquire the binary image 83 as the analysis result by inputting theanalysis target image 78 to the image analysis apparatus 200B. The userof the image analysis apparatus 200B also can acquire the cell nucleusregion weighted image 84 as the analysis result.

According to the image analyzing apparatus 200B according to the secondembodiment, the user can use the type of tissue selected by the user asa tissue for learning. This means that the learning of the neuralnetwork 50 is not left to the vendor side, and the user himself canimprove the degree of learning of the neural network 50.

Third Embodiment

Hereinafter, the image analysis system according to a third embodimentwill be described with respect to points different from the imageanalysis system according to the second embodiment.

Structure Summary

Referring to FIG. 14, the image analysis system according to the thirdembodiment includes a vendor side apparatus 100 and a user sideapparatus 200. The vendor side apparatus 100 operates as an integratedtype image analysis apparatus 100B and the user side apparatus 200operates as the terminal apparatus 200C. The image analysis apparatus100B is, for example, a general-purpose computer and is a device on thecloud server side that performs both of the deep layer learning processand the image analysis process described in the first embodiment. Theterminal apparatus 200C is composed of, for example, a general-purposecomputer, and is a user side terminal apparatus that transmits ananalysis target image to the image analysis apparatus 100B via thenetwork 99, and receives an image of the analysis result from the imageanalysis apparatus 100B via the network 99.

The image analysis system according to the third embodiment is similarto the image analysis system according to the second embodiment in thatthe integrated image analysis apparatus 100B installed on the vendorside has the functions of both the deep learning apparatus 100A and theimage analysis apparatus 200A according to the first embodiment. On theother hand, the image analysis system according to the third embodimentdiffers from the image analysis system according to the secondembodiment in that it is provided a terminal apparatus 200C and suppliesthe input interface of the analysis target image and the outputinterface of the analysis result image to the terminal apparatus 200C onthe user side. That is, the image analysis system according to the thirdembodiment is a client service type system in which the vendor sideperforms a deep learning process and an image analysis process andprovides a cloud service type of input/output interface for analysistarget images and analysis result images to the user side.

The image analysis apparatus 100B is connected to the imaging apparatus300 and acquires the first training image 70 and the second trainingimage 71 of the learning tissue captured by the imaging apparatus 300.

The terminal apparatus 200C is connected to the imaging apparatus 400,and acquires the analysis target image 78 of the analysis target tissuewhich is imaged by the imaging apparatus 400.

Hardware Structure

The hardware configuration of the image analysis apparatus 100B issimilar to the hardware configuration of the vendor-side apparatus 100shown in FIG. 5. The hardware configuration of the terminal apparatus200C is the same as the hardware configuration of the user apparatus 200shown in FIG. 6.

Function Block and Processing Procedure

Referring to FIG. 15, the processing unit 20B of the image analysisapparatus 200B according to the third embodiment includes a trainingdata generation unit 101, a training data input unit 102, an algorithmupdate unit 103, an analysis data generation unit 201, an analysis datainput unit 202, an analysis unit 203, and a cell nucleus regiondetection unit 204. These functional blocks are realized by installing aprogram that causes a computer to execute a deep learning process and animage analysis process in the recording unit 13 or the memory 22 of theprocessing unit 10B and executing this program by the CPU 11. The windowsize database 104 and the algorithm database 105 are recorded in therecording unit 13 or the memory 12 of the processing unit 10B, and bothare used jointly during deep learning and image analysis processing. Thelearned neural network 50 is stored beforehand in the algorithm database105 in association with the type of tissue, the coupling weight w isupdated by the deep learning process, and stored as the deep learningalgorithm 60 in algorithm data base 105.

The first training image 70 and the second training image 71 of thesample for learning are captured in advance by the imaging device 300and stored in advance in the recording unit 13 or the memory 12 of theprocessing unit 10B. The analysis target image 78 of the analysis targettissue is also imaged in advance by the imaging apparatus 400 andrecorded in the recording unit 23 or the memory 22 of the processingunit 20C of the terminal apparatus 200C in advance.

The processing unit 10B of the image analysis apparatus 100B performsthe process shown in FIG. 8 at the time of the deep learning process andthe processing shown in FIG. 11 at the time of the image analysisprocess. When describing each function block shown in FIG. 15, theprocesses of steps S11 to S13, S18 and S19 are performed by the trainingdata generating unit 101 during the deep learning process. The processof step S14 is performed by the training data input unit 102. Theprocesses of steps S15 to S17 are performed by the algorithm updatingunit 103. The processes of steps S21 and S22 are performed by theanalysis data generation unit 201 at the time of image analysis process.The processes of steps S23, S24, S26, and S27 are performed by theanalysis data input unit 202. The processes of steps S25 and S28 areperformed by the analysis unit 203. The process of step S29 is performedby the cell nucleus region detection unit 204.

The procedure of the deep learning process and the procedure of theimage analysis process performed by the image analysis apparatus 100Baccording to the third embodiment are similar to the proceduresperformed by the deep learning apparatus 100A and the image analysisapparatus 200A according to the first embodiment. Note that the imageanalysis apparatus 100B according to the third embodiment differs fromthe deep learning apparatus 100A and the image analysis apparatus 200Aaccording to the first embodiment in the following points.

The processing unit 10B receives the analysis target image 78 of theanalysis target tissue from the terminal apparatus 200C on the userside, and generates color density encoded diagrams 79 r, 79 g, and 79 bof R, G, and B colors from the received analysis target image 78 in stepS21 during the image analysis process shown in FIG. 11. The generationmethod of the color density encoded diagrams 79 r, 79 g, and 79 b is thesame as the generation method of step S11 at the time of the deeplearning process shown in FIG. 8.

In step S22 at the time of the image analysis process shown in FIG. 11,the processing unit 10B receives the input of the tissue type from theuser of the terminal apparatus 200C as the analysis condition throughthe input unit 26 of the terminal apparatus 200C. The processing unit10B refers to the window size database 104 and the algorithm database105 on the basis of the entered tissue type to set the window size usedfor analysis, and acquires the deep learning algorithm 60 used foranalysis.

In step S28 during the image analysis process, the processing unit 10Btransmits the binary image 83 of the analysis result to the terminalapparatus 200C on the user side. In the terminal apparatus 200C on theuser side, the processing unit 20C outputs the binary image 83 of thereceived analysis result to the output unit 27.

In step S29 during the image analysis process, the processing unit 10Balso performs a detection process of the region of the cell nucleus onthe binary image 83 of the analysis result following step S28. Theprocessing unit 10B creates a cell nucleus area weighted image 84 bysuperimposing the obtained cell nucleus region on the analysis targetimage 78 of the analysis target. The processing unit 10B transmits thecreated cell nucleus region weighted image 84 to the user terminalapparatus 200C. In the terminal device 200C on the user side, theprocessing unit 20C outputs the received cell nucleus region weightedimage 84 to the output unit 27, and ends the image analysis process.

As described above, the user of the terminal apparatus 200C can acquirethe binary image 83 as the analysis result by transmitting the analysistarget image 78 of the analysis target tissue to the image analysisapparatus 100B. The user of the terminal apparatus 200C also can acquirethe cell nucleus region weighted image 84 as the analysis result.

According to the image analysis apparatus 100B of the third embodiment,the user can be given the result of the image analysis process withoutacquiring the window size database 104 and the algorithm database 105from the depth learning apparatus 100A. In this way it is possible toprovide a service for analyzing the analysis target tissue and a servicefor discriminating the region of the cell nucleus as a cloud service.

The number of pathologists performing cytodiagnosis is insufficientnationwide. Pathologists are enrolled in a major hospital in urbanareas, but most are not in remote medical institutions or in relativelysmall medical institutions such as clinics even in urban areas. Thecloud service provided by the image analysis apparatus 100B and theterminal apparatus 200C aids tissue diagnosis and cytological diagnosisin such remote places or relatively small medical institutions.

Other Aspects

Although the present invention has been described in accordance with thesummary and specific embodiments, the present invention is not limitedto the above-described summary and specified embodiments.

Although the case of a gastric cancer has been described as an examplein the first to third embodiments described above, the sample to betreated is not limited thereto, and a sample of the above-describedtissue sample or a sample containing cells can be used.

Although the processing units 10A, 20B, and 10B refer to the window sizedatabase 104 to set the number of pixels of the window size in step S13in the first to third embodiments, the pixel number of the window sizealso may be directly set by the operator or the user. In this case, thewindow size database 104 is unnecessary.

Although the processing units 10A, 20B, and 10B set the number of pixelsof the window size based on the type of the input organization in stepS13 in the first to third embodiments, the size of the tissue also maybe input instead of inputting the type of tissue. The processing units10A, 20B, and 10B may set the number of pixels of the window size byreferring to the window size database 104 based on the size of the inputtissue. In step S22, as in step S13, the size of the tissue may be inputinstead of entering the type of tissue. The processing units 20A, 20B,and 10B may refer to the window size database 104 and the algorithmdatabase 105 to set the number of pixels of the window size and acquirethe neural network 60 based on the size of the input tissue.

Regarding the mode of entering the size of the tissue, the size may bedirectly input as a numerical value, or a user may input a predeterminednumerical range corresponding to the size to be selected and input bythe user, for example, using the input user interface as a pull-downmenu.

In steps S13 and S22, in addition to the type of the tissue or the sizeof the tissue, the imaging magnification at the time of capturing thefirst training image 70, the analysis target image 78, and the secondtraining image 71 also may be input. Regarding the mode of inputting theimaging magnification, the magnification may be directly input as anumerical value, or a user may select a predetermined numerical rangecorresponding to the magnification that the user intends to input, forexample, using the input user interface as a pull-down menu.

Although the window size is set to 3×3 pixels for the sake ofconvenience in the deep learning process and the image analysis processin the first to third embodiments, the number of pixels of the windowsize is not limited to this. The window size also may be set accordingto, for example, the type of the tissue sample and the type of thesample including cells. In this case, it suffices that the product ofthe number of pixels of the window size and the number of primary colorsincluded in the image corresponds to the number of nodes of the inputlayers 50 a and 60 a of the neural networks 50 and 60.

In step S13, the processing units 10A, 20B, and 10B also may acquire thenumber of pixels of the window size and correct the number of pixels ofthe acquired window size based on the input imaging magnification.

In step S17, the processing units 10A, 20B, and 10B record the deeplayer learning algorithm 60 in the algorithm database 105 in associationwith the organization type on a one-to-one basis in the first to thirdembodiments. Alternatively, in step S17, the processing units 10A, 20B,10B also may associate a plurality of tissue types with one deeplearning algorithm 60 and record them in the algorithm database 105.

In the first to third embodiments the hue is defined by a combination ofthree primary colors of light or a combination of three primary colorsof light, but the number of hues is not limited to three. The number ofhues also may be four primary colors plus yellow (Y) to red (R), green(G), and blue (B), or three primary colors of red (R), green (G), andblue (B) It may be a two primary color in which any one hue is reducedas two primary colors. Alternatively, one primary color of only one ofthe three primary colors of red (R), green (G), and blue (B) (forexample, green (G)) may be used. For example, the bright field images 70and the analysis target image 78 acquired using a known microscope, avirtual slide scanner or the like are not limited to color images ofthree primary colors of red (R), green (G), and blue (B), and may be acolor image of two primary colors or an image containing one or moreprimary colors.

Although the processing units 10A, 20B, and 10B generate the colordensity encoded diagrams 72 r, 72 g, and 72 b as single color images ofthree primary colors in step S11 in the first to third embodiments, thegradation of the primary colors of the color density encoded diagrams 72r, 72 g, and 72 b is not limited to 3 gradations. The gradation of thecolor density encoded diagrams 72 r, 72 g, and 72 b may be an image oftwo gradations or may be an image of one gradation or more. Similarly,although the processing units 20A, 20B, and 10B generate single-colorimages for each primary color of the color density encoded diagrams 79r, 79 g, and 79 b in step S21, the gradation of the primary color is notlimited to 3 gradations when generating the color density encodeddiagram. The primary color when creating the color density encodeddiagram may be an image of two gradations or may be an image of one ormore gradations. Illustratively, the gradation of the color densityencoded diagrams 72 r, 72 g, 72 b, 79 r, 79 g, 79 b can be set to 256levels (8 gradations) with color density values from value 0 to value255.

Although the processing units 10A, 20B, and 10B generate R, G, B colordensity encoded graphics 72 r, 72 g, and 72 b from the input firsttraining image 70 in step S11 in the first to third embodiments, theinput first training image 70 may be gradated in advance. That is, theprocessing units 10A, 20B, and 10B may directly obtain the color densityencoded diagrams 72 r, 72 g, 72 b of R, G, B colors from, for example, avirtual slide scanner or the like. Similarly, although the processingunits 20A, 20B, and 10B generate the color density encoded diagrams 79r, 79 g, and 79 b of the respective colors of R, G, and B from the inputanalysis target image 78 in step S21, the input analysis target image 78also may be gradated in advance. That is, the processing units 20A, 20B,and 10B may directly obtain the color density encoded diagrams 79 r, 79g, and 79 b of R, G, B colors from, for example, a virtual slide scanneror the like.

In the first to third embodiments described above, RGB is used for thecolor space when generating the color density encoded diagrams 72 and 79from the first training images 70 and 78 of color, however, the colorspace is not limited to RGB. In addition to RGB, various color spacescan be used such as YUV, CMY, and CIE L*a*b*.

In the first to third embodiments, density values of each pixel arestored in the order of red (R), green (G), and blue (B) in the trainingdata 74 and the analysis data 80, however, the order of storing andhandling density values is not limited to this. For example, the densityvalues may be stored in the order of blue (B), green (G), and red (R),and the order of arrangement of density values in the training data 74and the order of arrangement of density values in the analysis data 80may be the same.

Although the processing units 10A, 20B, and 10B binarize the gradationof each pixel of the input second training image 71 to generate the truevalue image 73 in step S12 of the first to third embodiments, it is alsopossible to acquire the binarized true value image 73 in advance.

Although the processing units 10A and 10B are realized as an integrateddevice in the first to third embodiments, the processing units 10A and10B need not be integrated devices, and may be any of a CPU 11, a memory12, a recording unit 13 and the like arranged in different locations andconnected via a network. The processing units 10A and 10B, the inputunit 16, and the output unit 17 are not necessarily arranged in oneplace, and they may be arranged separately from each other and connectedto each other so as to communicate with each other via a network. Theprocessing units 20A, 20B, 20C are also the same as the processing units10A, 10B.

Although each function block of the training data generation unit 101,the training data input unit 102, the algorithm update unit 103, theanalysis data generation unit 201, the analysis data input unit 202, theanalysis unit 203, and the cell nucleus region detection unit 204 isexecuted by a single CPU 11 or a single CPU 21 in the first to thirdembodiments, these function blocks are not necessary executed on asingle CPU, and also may be distributedly executed among a plurality ofCPUs. Each of these functional blocks also may be distributedly executedby a plurality of GPUs, or may be distributedly executed by a pluralityof CPUs and a plurality of GPUs.

In the second and third embodiments described above, programs forperforming the process of each step described in FIGS. 8 and 11 arerecorded in the recording units 13 and 23 in advance. Alternatively, theprogram may be installed in the processing units 10B and 20B from acomputer readable non-transitory tangible recording medium 98 such as aDVD-ROM or a USB memory. Alternatively, the processors 10B and 20B maybe connected to the network 99, and the program may be downloaded andinstalled from, for example, an external server (not shown) via thenetwork 99.

In the first to third embodiments, the input units 16 and 26 are inputdevices such as a keyboard or a mouse, and the output units 17 and 27are realized as a display device such as a liquid crystal display.Alternatively, the input units 16 and 26 and the output units 17 and 27may be integrated and realized as a touch panel type display device.Alternatively, the output units 17 and 27 may be composed of a printeror the like, and the binary image 83 of the analysis result or the cellnucleus region weighted image 84 of the cell nucleus may be printed andoutput.

Although the imaging device 300 is directly connected to the depthlearning device 100A or the image analysis device 100B in the first tothird embodiments, the imaging apparatus 300 also may be connected viathe network 99 to the deep learning apparatus 100A, or may be connectedto the image analyzing apparatus 100B. Similarly, although the imagingapparatus 400 is directly connected to the image analysis apparatus 200Aor image analysis apparatus 200B, the imaging apparatus 400 also may beconnected to the image analysis apparatus 200A or the image analysisapparatus 200B via the network 99.

EXAMPLE

Examples of the present invention will be shown below, and features ofthe present invention will be clarified.

Example 1

A deep learning process and image analysis process were performed in thestand-alone type system shown in the second embodiment. The tissues tobe learned and analyzed were gastric cancer tissues. Analysis wasperformed on two different gastric cancer tissue samples.

Creation of Training Data and Learning

Whole slide images (WSI) of bright field images of stomach cancer tissuestained with HE and fluorescent images of stomach cancer tissues stainedwith DAPI were color imaged using a virtual slide scanner. The imagingmagnification was 40 times. Thereafter, the color density values of eachof the R, G, B colors were gradated based on the bright field image, andcolor density encoding diagrams of each color of R, G, B were prepared.Based on the DAPI stained fluorescence image, a binary image was createdby binarizing the color density values of the region of the cell nucleusand the other region using a preset threshold value. FIGS. 16A and 16Bshow the bright-field image and the fluorescence image obtained byimaging, respectively, and the binarized image created from thefluorescence image is shown in FIG. 16C.

After that, training data were prepared by combining the color densityencoded diagram and the binarized image. The created training data weredivided into a window size of 113×113 pixels and the neural networklearned the divided training data of the window size as the input layer.For example, the 113×113 pixels adopted as the window size is a sizethat supports including in the window the entire shape of the cellnucleus region of at least one cell among the plural cells of about twoto nine cells.

Analysis Target Image Preparation

Similarly to the training data, a whole slide image of a bright fieldimage of HE-stained gastric cancer tissue was color-imaged using avirtual slide scanner. The imaging magnification was 40 times.Thereafter, color density encoded diagrams of each color of R, G, and Bwere created based on the captured bright field image, and an analysistarget image was created by combining the color density encoded diagramsof each of the created R, G, B colors.

Analysis Result

Analysis data with a window size of 113×113 pixels were created aroundeach pixel of the analysis target image and analysis data for thecreated window size were input to the learned neural network. Based onthe analysis results output from the neural network, the analysis datawere classified into a cell nucleus region and other region, and thecontour of the cell nucleus region is surrounded by white. The analysisresults are shown in FIGS. 17A-17B and 18A-18B.

FIGS. 17A-17B show the analysis result of the first stomach cancertissue sample image. FIG. 17A is a bright field image obtained bystaining the stomach cancer tissue with HE and FIG. 17B is an imageshowing a contour of the region of the cell nucleus obtained by theanalysis processing displayed superimposed on the image of FIG. 17A. InFIG. 17B, the region surrounded by white is the region of cell nucleus.

FIGS. 18A-18B show the analysis result of the second gastric cancertissue sample image. FIG. 18A is a bright field image obtained bystaining the gastric cancer tissue with HE, and FIG. 18B is an imageshowing a contour of the region of the cell nucleus obtained by theanalysis processing displayed superimposed on the image of FIG. 18A. InFIG. 18B, the region surrounded by white is the region of cell nucleus.

As shown in FIGS. 17A-17B and 18A-18B, it was possible to determinewhether a region is a cell nucleus region at an arbitrary position oftwo different types of pathological tissue images. The correctinterpretation rate of region determination of the cell nucleus was 85%or better.

Example 2

Cells imprinted with gastric tissue were subjected to Papanicolaoustaining and samples were prepared. With respect to this sample, thesame analysis processing as that in the first embodiment was performedusing the learned neural network. Cell imprinting was conducted on thegastric cancer site and the non-gastric cancer site. The analysisresults are shown in FIGS. 19A-19B and 20A-20B.

FIGS. 19A-19B show the analysis result of the touch imprint sample ofthe stomach cancer part. FIG. 19A is a bright field image obtained bystaining the gastric cancer tissue with HE, and FIG. 19B is an imageshowing a contour of the region of the cell nucleus obtained by theanalysis process displayed superimposed on the image of FIG. 19A. InFIG. 19B, the region surrounded by white is the region of cell nucleus.

FIGS. 20A-20B show the analysis result of the touch imprint sample ofthe gastric cancer site. FIG. 20A is a bright field image obtained bystaining the gastric cancer tissue with HE, and FIG. 20B is an imageshowing a contour of the region of the cell nucleus obtained by theanalysis process displayed superimposed on the image of FIG. 20A. InFIG. 20B, the region surrounded by white is the region of cell nucleus.

As shown in FIGS. 19A-19B and FIGS. 20A-20B, it was possible todetermine whether a region is a cell nucleus region at an arbitraryposition of a touch imprinted sample which is different than thestaining mode from the above-mentioned Example 1.

What is claimed is:
 1. An image analysis method for analyzing an imageof a tissue or a cell to be analyzed using a deep learning algorithm ofa neural network structure, the method comprising: generating aplurality of analysis data from an analysis target image that include apart of the tissue or a part of the cell to be analyzed, the pluralityof the analysis data being generated for each region having apredetermined number of pixels relative to one analysis target image;inputting the plurality of analysis data to the deep learning algorithm,and generating data indicating a region of a cell nucleus in theanalysis target image by the deep learning algorithm, the dataindicating the region of the cell nucleus distinguishingly from anotherregion or indicating a boundary between a region of the cell nucleus andanother region.
 2. The image analysis method according to claim 1,wherein the analysis target image is an image of a tissue diagnosticsample, and the analysis target image includes a hue comprised of oneprimary color or a hue obtained by combining two or more primary colors.3. The image analysis method according to claim 1, wherein the analysistarget image is an image of a cell diagnostic sample, and the analysistarget image includes a hue comprised of one primary color or a hueobtained by combining two or more primary colors.
 4. The image analysismethod according to according to claim 1, wherein the deep learningalgorithm determines whether an arbitrary position in the analysistarget image is a region of a cell nucleus.
 5. The image analysis methodaccording to claim 1, wherein analysis data are generated for eachregion of the predetermined number of pixels including peripheral pixelscircumscribing a predetermined pixel; and the deep learning algorithmgenerates a label indicating whether the predetermined pixel is one ofthe region of the cell nucleus.
 6. The image analysis method accordingto claim 1, wherein a number of nodes of the input layer of the neuralnetwork corresponds to a product of the predetermined number of pixelsof the analysis data and a number of combined primary colors.
 7. Theimage analysis method according to claim 2, wherein the tissuediagnostic sample is a stained sample, and the analysis target image isan image obtained by imaging the stained sample under a bright fieldmicroscope.
 8. The image analysis method according to claim 2, whereinthe cell diagnostic sample is a stained sample, and the analysis targetimage is an image obtained by imaging the stained sample under a brightfield microscope.
 9. The image analysis method according to claim 7,wherein a stain for bright-field observation compriseshematoxylin-eosin.
 10. The image analysis method of claim 8, wherein astain for bright field observation is Papanicolaou.
 11. The imageanalysis method according to claim 1, wherein the deep learningalgorithm classifies the analysis data into classes indicating a regionof a cell nucleus contained in the analysis target image.
 12. The imageanalysis method according to claim 1, wherein an output layer of theneural network comprises a node having a softmax function as anactivation function.
 13. The image analysis method according to claim 1,further comprising outputting a cell nucleus region weighted image thatthe region of the cell nucleus is superimposed on the analysis targetimage.
 14. An image analysis apparatus for analyzing an image of atissue or a cell using a deep learning algorithm of a neural networkstructure, the apparatus comprising a system configured to: generating aplurality of analysis data from an analysis target image that include apart of the tissue or a part of the cell to be analyzed, the pluralityof the analysis data being generated for each region having apredetermined number of pixels relative to one analysis target image;inputting the plurality of analysis data to the deep learning algorithm,and generating data indicating a region of a cell nucleus in theanalysis target image by the deep learning algorithm, the dataindicating the region of the cell nucleus distinguishingly from anotherregion or indicating a boundary between a region of the cell nucleus andanother region.
 15. An image analysis method for analyzing an image of atissue or a cell to be analyzed using a deep learning algorithm of aneural network structure, the method comprising: storing a plurality ofdeep learning algorithms for a specific type of a sample including atissue or a cell, the specific type being different among the pluralityof deep learning algorithms; generating analysis data from an analysistarget image that includes the tissue or the cell to be analyzed;selecting one of the deep learning algorithms depending on a type of asample including the tissue or the cell to be analyzed; inputting theanalysis data to the selected deep learning algorithm, and generatingdata indicating a region of a cell nucleus in the analysis target imageby the deep learning algorithm.
 16. The image analysis method accordingto claim 15, wherein the type of a sample is inputted via an inputdevice.
 17. The image analysis method according to claim 15, furthercomprising setting a size of the analysis data inputted to the deeplearning algorithm at a time of one input depending on the type of asample.
 18. The image analysis method according to claim 15, wherein thetype of a sample comprises a sample including a gastric tissue.
 19. Theimage analysis method according to claim 15, wherein the deep learningalgorithms have been trained depending on the type of a sample.
 20. Animage analysis apparatus for analyzing an image of a tissue or a cellusing a deep learning algorithm of a neural network structure, theapparatus comprising a system configured to: storing a plurality of deeplearning algorithms for a specific type of a sample including a tissueor a cell, the specific type being different among the plurality of deeplearning algorithms; generating analysis data from an analysis targetimage that includes the tissue or the cell to be analyzed; selecting oneof the deep learning algorithms depending on a type of a sampleincluding the tissue or the cell to be analyzed; inputting the analysisdata to the selected deep learning algorithm, and generating dataindicating a region of a cell nucleus in the analysis target image bythe deep learning algorithm.